Shadow Theory, data model design for data integration

In information ecosystems, semantic heterogeneity is known as the root issue for the difficulties of data integration, and the Relational Model is not designed for addressing such challenges, i.e. to re-use data that is modeled from other sources in the local data model design. Although researchers have proposed many different approaches, and software vendors have designed tools to help the data integration task, it remains an art relying on human labors. Due to lacks of a comprehensive theory to guide the overall modeling process, the quality of the integrated data heavily depends on the data integrators’ experiences. Based on observations of practical issues for enterprise customer data integration, we believe that the needed solution is a new data model. This new data model needs to manage the difficulties of semantic heterogeneity: different data collected from different perspectives about the same subject matter can naturally be inconsistencies or even in conflicts. In other words, existing data models are designed to support single version of the truth, and naturally they will have difficulties during data integration when data collected from different sources are based on different perspectives or at different levels of abstraction. Therefore, we propose Shadow Theory to serve as the philosophical foundation in order to design a new data model. The kernel of the theory is based on the notion of shadow, which can be traced back to Plato’s Allegory of the Cave over 2000 years ago. The basic idea is that whatever we can observe and store into databases about the subject matter are just shadows. Meanings of shadows are mental entities that exist only in the viewers’ cognitive structures. Such mental entities are constrained by the viewers’ internal model about the reality, especially the implicitly or explicitly chosen perspective(s) or ontology, if formally represented. In this paper, we propose six basic principles to guide the overall data model design. Further, we also propose algebra with a set of basic operators to support data operations by their meanings, not by their logical structure. The representation is based on pointfree geometry such that any meaning is represented as an area in semantic space, which can be decomposed or aggregated in different ways concurrently. We use W(hat)-tags to attach on shadows for their meanings, and E(quivalence)-tags to recognize what meanings can be treated as the same. We use enterprise customer data integration as an example to illustrate the data model design and operation principles.